Learning by Online Gradient Descent

We study online gradient{descent learning in multilayer networks analytically and numerically. The training is based on randomly drawn inputs and their corresponding outputs as deened by a target rule. In the thermo-dynamic limit we derive deterministic diierential equations for the order parameters of the problem which allow an exact calculation of the evolution of the generalization error. First we consider a single{layer perceptron with sigmoidal activation function learning a target rule deened by a network of the same architecture. For this model the generalization error decays exponentially with the number of training examples if the learning rate is suuciently small. However, if the learning rate is increased above a critical value, perfect learning is no longer possible. For architectures with hidden layers and xed hidden{to{output weights, such as the parity and the committee machine, we nd additional eeects related to the existence of symmetries in these problems.